Method For Determining A Video Capture Interval For A Calibration Process In A Multi-Projector Display System

ABSTRACT

First, a delivery delay, i.e. the time it takes for a content to be captured and a representative frame made available (i.e. delivered), is determined. Second, a video capture interval, i.e. a minimum time delay requirement between stable captured content frames, is determined. In other words, the recycle time-period after which it is safe to change frame content during a capture interval without impacting a prior stable video frame in the pre-delivery stage is determined.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to commonly owned U.S. patent applicationSer. No. 12/728,838, filed Mar. 22, 2010, and Ser. No. 12/549,011, filedAug. 27, 2009, which are hereby incorporated by reference in theirentirety.

FIELD OF INVENTION

The present invention relates generally to multi-projector displaysystems. More particularly, the present invention relates to accuratecalibration of the system during registration of each of the multipleprojectors.

BACKGROUND

With an ever growing desire to see more information with better quality,large-screen displays have become quite popular. Increasing screenresolutions and sizes are continually emerging and made available intelevisions, computer monitors, and other video devices. Until recently,large screen displays were typically too costly, physically unwieldy, orsimply unavailable. Video projectors provided one solution, enabling awide range of communication and entertainment functions by offering asignificantly greater display area relatively inexpensively. Thesedevices have found application in conference rooms for presentations,home theaters, classroom training, and advertising billboard displays.

Similar to other video device technologies, video projectors continue toadvance their displayable pixel resolution and light output. Todaycommodity projectors are brighter, offer better quality, and are oftenless expensive than those of prior years. Highly portable projectors (inboth weight and size) are also becoming readily available. No longer docommodity projectors remain constrained to dimly lit rooms with wellprepared display surfaces. A video projector's small physical sizerelative to its large projection output therefore remains appealing.

Even with these improvements, however, it is still difficult orimpossible for a single commodity projector to achieve very highresolutions, project over vast areas, or create bright projections onvery bright surface areas (for example, near day lit windows).Applications demanding such display qualities, however, are becomingmore desirable. The benefits of increased resolution, brightness, andlarger display surface area have proven useful for reaching largeraudiences and providing full-scale life-sized immersive environments.Unfortunately, construction of such large displays is complex andcostly.

One common technique, such as grouping multiple projectors together andtiling their projection output to produce large screen displays of anydesired size, presents challenging problems with registration (that is,alignment of projector pixels). Color and luminosity variance acrossseparate devices and even within a given device is difficult to correct.Minor shape or geometric inconsistencies of the display surface can alsohinder adequate results. Projector lumens, or light output, may not beadequate for brighter locations. Synchronizing content delivery to theindividual components forming the larger display are additional hurdlesto solve. Some of these problems apply to single-projector displays aswell.

Solutions to some of these system problems take many forms. Many requireprecise pixel and color alignment using manual methods that requirephysical adjustment of the projector placement. If the output pixelsfrom one projector are not close enough to those from another projector,a visible gap may occur between the projections on the compositedisplay. Likewise, overlapping pixels across projectors produce brightseams that are also objectionable. High-end projectors with specializedlens optics or edge blending/blurring filters may be available to reducesome of these problems, but are far from optimal.

Specialized projectors and mounting hardware, measurement tools, andtedious calibration methods are additional requirements that add to theresource costs and complexities of physical projector alignment whichcan become too demanding for the average user. The advanced skills andtime requirements are more than most will invest. In manyconfigurations, physical alignment may even be impossible usingprojectors with limited optic pathways, or with even slightly irregulardisplay surfaces. When changes are necessary to replace failed lamps,the calibration methods often need repeating.

What is needed is a system that provides an easy calibration andplayback mechanism offering the typical user an automated method tocreate a composite display from one or more commodity projectors, evenin the presence of high ambient light levels. This method should offer arelatively quick one-time calibration function to be performed onceafter casual projector placement or changes. Commonly owned U.S. patentapplication Ser. No. 12/728,838, entitled “Multi-projector displaysystem calibration” describes such a one-time calibration system.

Digital still cameras are often used by multi-projector display systems(such as the one described in U.S. patent application Ser. No.12/728,838) to gather input for a calibration process. These cameras cancapture high-resolution images of the environment for detailedmeasurement and analysis. Exposure settings may be accurately configuredand the capture event (exposure) can be directly initiated or otherwisecontrolled by remote shutter signaling or the like. The moment ofexposure is thus deterministic (or occurring after shutter signaling)and the single received frame result marks the end of a captureinterval. Therefore, a process requiring many exposures of variedcontent can be easily and optimally automated, as in a loop: prepare acontent, capture a content, receive captured result, repeat.

Digital still cameras, however, may exhibit slow recycle times betweenexposures. Recycle time may be slowed by internal processing ofhigh-resolution image content or transfer and data storage of largeframe data. This can diminish the performance of a calibration capturefunction in a calibration processing step. Additionally, it decreasesthe capability to quickly position a camera properly, toward acalibration display, as the time to gather multiple frames and determinean acceptable orientation of the camera's view is lengthened.

In a multi-projector system (such as the one described in U.S. patentapplication Ser. No. 12/728,838), an inexpensive webcam is used insteadof a more costly digital still camera. Higher frame rates often enableuser interfaces with video feedback to aide positioning of the cameraand aligning its view with a display surface. Lower device resolutionsmay lead to diminished storage requirements. Because the video framerate can be fast, it is possible to shorten the time for a calibrationtest pattern capture process. However, using a video device to capturecalibration content presents several challenges. With a webcam, there isgenerally less control of exposure settings and exposure notificationcompared with a still camera. Limited methods to adjust an aspect of anexposure sensitivity setting may indirectly change a video frame rate ora noise characteristic. Capture methods tend to require more complexinterfaces like asynchronous delivery of a continuous sequence, orstream, of captured frames.

Some video cameras and webcams are configurable to capture still imagesby using internal resolution enhancement or frame averaging methods todeliver higher resolution frames. If the quality and performance isacceptable for a calibration process, then the still image interfacesmay be used.

However, when video devices are used and a contiguous stream of capturedframes is arriving at a calibration process, one key challenge isdetermining when a delivered frame will contain a known element capturedby the device. Additionally, it is difficult to determine when a changeto a known element for capture by the device can be made withoutimpacting a previously captured element in an as yet to be deliveredframe.

For example, a calibration process requires that views of severalcalibration test patterns be captured and stored for future processing.The process begins with outputting a calibration test pattern onto adisplay surface. With a still camera, a signaling event sent after thecalibration test pattern is displayed begins the capture operation.After the exposure event (when the camera captured content is determinedto be complete through notification, detection of delivery to a storagemedia, transmission from the camera to the calibration process, or aknown exposure interval), the process continues for a next calibrationtest pattern. With a webcam, there is no exposure signaling. The videocapture process is started (or already active) and it is initiallyunknown which frame in a delivery sequence contains the expected captureof the calibration test pattern.

Contributing to the delivery delay and uncertainty are variablesincluding time to enable a video capture process, frame buffers in thedevice or in a driver, data compression or decompression, transmissiontime, slow gain or frame averaging methods, and other delays between theactual frame capture event and the location of frame access, etc. It isequally unclear when to display a next calibration test pattern aschanges to display content mid-frame (i.e. at the interval of videoframe capture), can yield unexpected results within the frame.

Accurate frame rate and timestamp information may not be available fromthe webcam device. Even if a frame rate is well controlled andidentified, it alone is insufficient to determine the frame delaybetween a capture time and a delivery time or the time between asignaling to start and the actual start of a capture stream. Frametimestamps, if available and associated with a capture process clock,can help to determine the delivery delay, but cannot be used alone todetermine how quickly capture content can be safely changed.

SUMMARY OF INVENTION

The present invention is directed to the problems described above.First, the present invention determines a delivery delay interval, thetime it takes for a projected content to be captured and arepresentative frame made available. Second, it determines a minimumtime delay requirement between stable captured content frames, i.e. avideo capture interval. In other words, the recycle time-period afterwhich it is safe to change projected content during a capture intervalwithout impacting a prior stable video frame in the pre-delivery stageis determined.

As a component of a calibration process in a multi-projector displaysystem, the present invention determines a delay interval betweendisplay of a detection pattern and retrieval of a video frame containinga captured view of the detection pattern from a capture device videostream and where the delay interval is a minimum. The present inventionfurther determines a video capture interval which specifies the minimumtime needed for a capture device to stabilize a change in a captureddetection pattern onto a video frame. These intervals are used by acalibration process to capture views of a set of calibration testpatterns.

Other objects and attainments together with a fuller understanding ofthe invention will become apparent and appreciated by referring to thefollowing description and claims taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings wherein like reference symbols refer to like parts:

FIG. 1 is a general block diagram of the system of the presentinvention;

FIG. 2 is a functional overview of the present invention;

FIG. 3 is a general flow diagram of the method of the present invention;

FIG. 4 illustrates a sample timeline calibration capture processhistory;

FIG. 5 illustrates an interval detection process history; and

FIG. 6 illustrates a video capture frame change frequency.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates to camera-based registration of projectordisplay systems. During registration, one or more projectors projectcalibration images upon a display surface. A camera captures one or moreimages of each projected calibration image. A projector platform createsa mapping, for each projector, between pixel locations of the camera andpixel locations of the projector. After registration, the mappings areused to alter the images prior to projection so that seamless compositeimage can be displayed by the multiple projectors.

These mappings are stored in Virtual Pixel Map (VPM) files that form anaccurate pixel registration map of each projector. Using the VPM files,each physical device pixel on a projector can be mapped to one or morelogical pixels in a virtual region defining the large format display(composite image). This virtual large format display resolution definesthe unit of each VPM pixel, which is a virtual pixel in the logicaldisplay region. It also defines the relative location of VPM pixels onone device relative to VPM pixels on another. The bounded display regionrepresents a homogenous portion of the logical display region withinwhich every pixel is addressable by at least one physical device pixel.

In order to project what appears to be a single continuous image, eachindividual projector's intrinsic and extrinsic parameters must all be ina common coordinate frame. This enables multiple projectors to generatea rectangular image of known aspect ratio. Projector platform 106,discussed below, performs registration of the projectors to create anaccurate pixel registration map of each projector.

The registration process can use one or both of two techniques to createthe mappings. According to the first technique, a coarse mapping iscreated for each projector using striped calibration images, which canbe generated using Gray codes. This technique can establish a mapping towithin approximately sixteen pixels. According to the second technique,raster calibration images are used. When both techniques are used, aprecise mapping can be created between pixel locations of the camera andpixel locations of the projector. The mappings can be used forcorrecting projection distortions such as keystoning, aligning multipleprojections, and the like. The first technique of using Gray codes isdiscussed in commonly assigned U.S. patent application Ser. No.12/728,838, filed Mar. 22, 2010, which is hereby incorporated byreference in its entirety. The second technique is discussed in commonlyassigned U.S. patent application Ser. No. 12/549,011, filed Aug. 27,2009, which is hereby incorporated by reference in its entirety.

FIG. 1 shows a multi-projector display system 100 according to anembodiment of the present invention. System 100 includes four projectors102A-102D aimed at a display surface 104. Of course, other numbers ofprojectors 102 can be employed. For example, two projectors 102 can beused. Data is provided to projectors 102 by a projector platform 106,which can obtain source input from a media player 108, a computer 110, asource connected by a network 112 such as the Internet, and the like.For calibration, system 100 includes a video camera 114 (e.g. a webcam).

In an embodiment, system 100 includes four projectors 102, and aprojector platform 106 that is implemented as a personal computer (PC)configured with a central processing unit (CPU) and graphic processingunits (GPU) providing four video outputs, each connected to one ofprojectors 102. An optional capture card provides video input fromsources such as computer 110, media player 108, and the like. A videocamera 114 is attached to the PC for the calibration process. In otherembodiments, a digital still camera may be used for the calibrationprocess. After calibration, camera 114 may be removed or used as a mediainput device by projector platform 106.

Projectors 102A-102D produce respective component projections 120A-120Dupon display surface 104. Together component projections 120A-120D forma single composite projection 122. Note that component projections 120overlap in regions 124A-124C.

Unless specified otherwise, the method or steps discussed with respectto the present invention are performed by the projector platform 106,which as mentioned above, is implemented as a personal computer (PC)configured with a central processing unit (CPU) and graphic processingunits (GPUs). One or more of the processing units or “processors” isused to execute the method or steps under control of software orfirmware stored in memory in the PC or in an external memory such ascomputer-readable media.

As mentioned above, the registration process can use one or both of twotechniques to create the mappings. A detailed discussion of thesetechniques is not required to understand or use the present invention.However, these techniques are described in detail in commonly owned U.S.patent application Ser. No. 12/728,838, filed Mar. 22, 2010, and Ser.No. 12/549,011, filed Aug. 27, 2009, which are hereby incorporated byreference in their entirety.

A calibration capture process includes a video camera 114 configured tocapture a set of calibration test patterns each projected sequentiallyonto a display surface 104. Captured frames associated with eachcalibration test pattern are required by a future calibration processingstep. The count of calibration test patterns and a rate of capturedetermine the duration of the calibration capture process. Ideally, thisduration is a minimum. Finding the parameters that can be used toapproach this minimum is one goal of the present invention. Theseparameters include a determination of a delivery delay interval and avideo capture interval.

In general terms, a capture process is a loop: project a pattern, sample(capture) the pattern, store the sample result, and repeat for allpatterns. A digital still camera is one device that can capture andprovide a single representative sample (frame) when requested. However,video devices provide a continuous stream of frames and introduce acontinual frame video capture method asynchronous to the capture processloop. This is often implemented in a separate processing thread. It istherefore unclear when a sample (frame) is available which is arepresentative sample, associated with a projected pattern, and providedfor storage, and thus when a new pattern can be projected in the captureprocess loop.

In an embodiment of a calibration process, a calibration capture processis coupled “synchronously” to the asynchronous video capture methodusing a delivery delay interval as determined by the present invention.In this embodiment, the calibration capture process projects acalibration test pattern, waits for a delivery delay interval, storesthe latest available frame from a video capture method as the associatedsample, and repeats the process for all calibration test patterns.

In another embodiment of a calibration process, a calibration captureprocess is coupled “asynchronously” to the asynchronous video capturemethod using a delivery delay interval and a video capture interval asdetermined by the present invention. In this embodiment, the calibrationcapture process is divided into two simultaneously executing components.A first component projects a calibration test pattern and places an itemcontaining the display time (DT) and an identification labeling theassociated calibration test pattern (id CTP) into a component sharedcalibration capture queue.

The first component continues by waiting a video capture interval beforerepeating the first component process for all calibration test patterns.A second component waits for items to arrive into the component sharedcalibration capture queue. When an initial item arrives (the first itemreceived after the second component has started executing), the secondcomponent begins a one time initialization step which waits until theitem's time DT+delivery delay interval before continuing. On subsequentitem arrival, the second component waits until the item's time DT+videocapture interval before continuing. The second component then startsstoring the latest available frame from a video capture method as anassociated sample to a calibration test pattern as indicated by theitem's CTP. The second component repeats the process and returns to waitfor a subsequent item to arrive.

In this method, a display surface 104 is continuously captured by avideo device 114. The device capture or sampling rate, the deliveryframe rate, and the exposure duration of each frame are a function ofthe video device configuration and are not always known. As patterns areprojected onto a display surface 104, their image is captured by thevideo device 114. It may take some time before captured frames from thevideo capture method containing a view of the projected pattern becomeavailable to a capture process. This method determines this time.Likewise, changing a pattern during continuous video capture may produceincomplete or underexposed views of the calibration test pattern for oneor more available video frames. This method ensures that enough delayexists in the process such that frames retrieved from the video capturemethod and after the delay are best representative of the displaypattern and do not exhibit such error.

In this method, the video device capture time or frame numberrelationship between an exposure event and the associated deliveredframe is initially unknown. Temporal video frame averaging may beemployed in a camera driver that smoothes changes in a scene (typicallyused to reduce noise) but can result in a reduced delivery frame rateindependent of the device sample rate. Similarly, increasing an exposurelevel may result in a change to the available frame delivery frequency.These delays can also impact the rate at which calibration test patternsmay be gathered.

A goal is to determine an optimal minimal time with which to wait beforerecording an available captured frame associated with a calibration testpattern and furthering a continuing calibration capture process byupdating to a next calibration test pattern. This optimal video captureinterval is determined such that a frame grabber process can prepare torecord a video frame located at a future time or frame count from acurrent time or frame counter which will be associated with acalibration capture image projected at the current time or framecounter. The main calibration process may continue with updating orprojecting another calibration test pattern at a determined frequencyindependent of the actual frame storage with confidence that theupdating will not affect the as yet to be recorded frame result from aprior calibration test pattern.

FIG. 2 is a functional overview of the present invention. Certainfunctional blocks can be hardware or software or combinations of both,such as projector platform 106 (as a PC) executing program instructions.FIG. 3 is a general flow diagram showing the major steps in the methodof the present invention.

The method of present invention method begins by projecting a series ofcalibration test patterns from one or more projectors 1 to N (i.e.projectors 102A to 102D in FIG. 1) onto screen 104 and initiatingcapture by a video device (webcam 114) and receiving a capture stream(FIG. 3, step 310). For example, on a Microsoft Windows platform awebcam device 114 may be accessed through a Microsoft DirectX capturefilter graph. This filter graph is built and configured with a framegrabber filter interface. Referring to FIG. 2, the frame grabber filter210 is configured to facilitate retrieval of individual video frames 212collected from the capture stream 214. The frame grabber filter 210maintains a time tFn and a counter cFn associated with all framesreceived. This video capture method (shown as functional block 216)captures and collects frames from the webcam 114, keeping a latest frame212 for use by a gathering method, but disposing of it and any unusedframes as a new frame becomes available.

Next, the method identifies of a region within a captured video framewhose pixels contain a sample portion or view of the projected displaysurface, which is labeled a region of detection 218 (step 312 in FIG.3). This may be accomplished through a user interface selection, aswithin a preview interface for camera alignment, or similar operation.In a preferred embodiment, the region of detection 218 is a rectangularshaped area with normalized position and dimension units so it can beautomatically applied to other captured video frames. It will beappreciated that projector platform 106 manipulates the projectedoutput, and thus the pixel content in the region of detection 218 incaptured frames will be varied. Identification of a region narrows apixel inspection range for determining pattern changes.

Delivery Delay Interval—Delay Detection

The method of the present invention continues with a delay detectionprocess 220, which is shown generally as step 314 in FIG. 3. First, aprojector 102 outputs a detection pattern with all pixel values having asubstantially high luminance component (for example, a maximum luminanceall white image) and projector platform 106 records the output timetHO_0. The frame most recently delivered and available to the videocapture method, the current video frame, is continually monitoredthroughout the process. The current video frame time tFn and counter cFnare also recorded as tHF_0 and cHF_0.

Next, a frame monitoring operation 222 is enabled that allows the framegrabber filter 210 to process each video frame received with aninspection of the assigned region of detection 218 within the capturedframe 212. Inspection includes identification, measurement of pixelswithin the assigned region, and evaluation of measurement results (shownas 224 in FIG. 2). Measurement may include computing a mean, an average,frame comparison, or other operation. Results of inspection are recordedwith the frame and the monitor operation.

For delay detection, inspection involves measuring a median among theregion pixel values and maintaining a short moving median amongsuccessive monitoring frames. A rate of change between thesemeasurements is computed. The highest median measured and the lowestmedian measured are also maintained, i.e. stored in memory. This can beperformed by simply storing the first median as the highest and lowestmedian value and then comparing each new median with the stored highestand lowest and then maintaining or replacing the appropriate highest orlowest stored median value as a result of the comparison.

For delay detection, evaluation monitors when the inspection measurementresults indicate the pixels within the region of detection in thecaptured frame “matches” the detection pattern of interest (i.e. asubstantially high luminance condition). Evaluation match is successfulwhen the rate of change measurement is low (e.g. below a smallthreshold, such as 10% of the change range), the region median is nearthe highest or lowest median measured (i.e. as a small distancepercentage within the range between the highest or lowest median and amidpoint between them, such as within 25% of the range from the medianto the midpoint), and the highest and lowest median values are separatedby at least a quarter of a luminance range (i.e. if output pixelsluminance can range between 0 and 255 units, then input measurementsshould be separated by at least 255/4 values) or a timeout conditionoccurs. The timeout condition may typically occur at the start of theprocess when the content captured before an initial detection patternclosely matches the initial detection pattern. The timeout may be a longinterval, such as 20 seconds, or some function of the available framerate, such as 10×FPS seconds. When the “match” is successful, theidentified current frame time tFn and counter cFn results are recordedfor the current detection pattern (i.e. tHR_0 and cHR_0).

Next, projector 102 outputs a detection pattern with all pixel valueshaving a substantially low luminance component (for example, a minimumluminance all black image) and records the output time tLO_0 along withthe current video frame time tFn and counter cFn as tLF_0 and cLF_0. Theenabled monitoring operation inspects for a match to this new detectionpattern (i.e. a minimum luminance condition) and again, when successful,the current frame time and counter results are stored as tLR_0 andcLR_0.

The delay detection process repeats for several iterations ‘i’,collecting and assembling results in a table [(tHO_0, tHF_0, tHR_0,cHF_0, cHR_0), (tHO_i, tHF_i, tHR_i, cHF_i, cHR_i)] and [(tLO_0, tLF_0,tLR_0, cLF_0, cLR_0), (tLO_i, tLF_i, tLR_i, cLF_i, cLR_i)]. A differencebetween the time of a received frame matching the pattern and detectionpattern output time indicates a measurement of delivery delay. Forexample, tHR_(—)0−tHO_(—)0 indicates a delivery delay. LikewisetLR_(—)0−tLO_(—)0 also measures the delay. Alternately, a delivery delaycan be determined as the difference of the frame counter values (i.e.cHR_i−cHF_i) divided by the delivery frame rate. The max delivery delayD(i) is computed with each iteration as a maximum of the individual Hiand Lo delivery delay calculations. Since the startup frame conditionsare initially unknown, a moving average of a most recent ‘m’ computedmax delivery delay results (i.e. D(i-m) through D(i)) is computed witheach iteration. When a change in the moving average approaches aminimum, the iteration process can stop.

The maximum max delivery delay result among the most recent ‘m’iterations is determined to be the delivery delay interval 226 (step 316in FIG.3). This ends the delay detection process 220.

Video Capture Interval—Change Detection

The method of the present invention continues with a process todetermine a video capture interval, which is not the same as a videocapture method frame rate. Initially, the video capture interval is setto an amount of time equal to (or greater than) the delivery delayinterval 226. With such setting, a capture method need only wait for anext frame delivery after a delay capture interval to continuingprocessing. Though this capture approach is possibly slower than anasynchronous approach which uses an optimal video capture interval, thesynchronous nature of a “wait until received” capture method may beeasier to implement in certain processes.

Typically a safe guess of a default video capture interval 228 isapproximately 2 times the output frame rate of the camera. This providesenough headroom to accommodate changes to display content withoutaffecting pixels in gathered frames. However, when a video camerautilizes frame averaging (i.e. to control noise in the frame image) orother processing that impacts individual frames in a temporal manner,individual frames may not be “stable” at a frame rate interval (sinceseveral frames may record some aspect of a captured content changebefore the frame changes are minimized). When this happens, the videocapture interval 228 may be much larger.

The method continues with a change detection process 230, which is showngenerally as step 318 in FIG. 3. In one embodiment, this process occursin tandem with the delay detection process 220 after a first maxdelivery delay result. In FIG. 2, delay detection process 220 and changedetection process 230 are shown as part of an overall detection captureprocess 238. As an initial one-time operation, projector 102 outputs adetection pattern with all pixel values having a substantially lowluminance component (for example, a minimum luminance all black image)and the process waits a delivery delay interval (or a max deliver delayresult if the delivery delay interval has not yet been established.)This ensures that any received frames available by the video capturemethod have a region of detection 218 with region pixels values at asubstantially low value. This one-time operation is unnecessary if thechange detection process 230 immediately follows the delay detectionprocess 220.

First (or following the one-time operation, if performed), projector 102outputs a detection pattern with all pixel values having a substantiallyhigh luminance component (for example, a maximum luminance all whiteimage) and records the output time tHO_0 along with the current videoframe time tFn and counter cFn as tHF_0 and cHF_0.

Next, the frame monitoring operation 222 is again enabled. For thechange detection process 230, inspection also measures a median amongthe region pixel values and maintains a short moving median amongsuccessive monitoring frames. A rate of change between thesemeasurements is computed. The highest median measured and the lowestmedian measured are also maintained. Evaluation, however, monitors fortwo events. The first is when measurement results indicate the captureframe region measurements are beginning to deviate from a priordetection pattern match. The second is when measurements resultsindicate the captured frame region “matches” the detection pattern ofinterest (same as with a delay detection process 220). Then theevaluation event detects the measurement result is “changing” from aprior detection pattern. To determine this change, an evaluationmeasures the rate of change between successive frame medianmeasurements. When the rate of change shows an increase (such as above10% of the change range), then a measurement result change event hasbeen detected. The current frame time tFn and count tFc results arerecorded for the current detection pattern as tHC_0 and cHC_0 at adetection change output time. When the evaluation event detects themeasurement result “matching” the detection pattern, the current frametime tFn and counter tFc results are recorded for the current detectionpattern (i.e. tHR_0 and cHR_0).

Next, projector 102 outputs a detection pattern with all pixel valueshaving a substantially low luminance component (for example, a minimumluminance all black image) and records the output time tLO_0 along withthe current video frame time tFn and counter cFn as tLF_0 and cLF_0. Theenabled monitoring operation awaits successful evaluations. When the“changing” evaluation is successful, the current frame time and counterresults are stored as tLC_0 and cLC_0. When the “matching” evaluation issuccessful, they are stored as tLR_0 and cLR_0.

Like the delay detection process 220, the change detection process 230repeats for several iterations ‘i’, collecting and assembling results ina table [(tHO_0, tHF_0, tHR_0, cHF_0, cHR_0, tHC_0, cHC_0), (tHO_i,tHF_i, tHR_i, cHF_i, cHR_i, tHC_i, cHC_i)] and [(tLO_0, tLF_0, tLR_0,cLF_0, cLR_0, tLC_0, cLC_0), (tLO_i, tLF_i, tLR_i, cLF_i, cLR_i, tLC_i,cLC_i)]. A difference between the time of a received frame matching thepattern and a detection change output time indicates a measurement ofchange interval. For example, tHR_(—)0−tHC_(—)0 indicates a changeinterval. Likewise tLR_(—)0−tLC_(—)0 also measures the change interval.Alternately, a change interval can be determined as the difference ofthe frame counter values (i.e. cHR_i−cHC_i) divided by the deliveryframe rate. The max change interval C(i) is computed with each iterationas a maximum of the individual Hi and Lo change interval calculations.Since the startup frame conditions are initially unknown, a movingaverage of a most recent ‘m’ computed max change interval results (i.e.C(i-m) through C(i)) is computed with each iteration. When a change inthe moving average approaches a minimum, the iteration process can stop.

The maximum max change interval result among the most recent ‘m’iterations is determined to be the video capture interval (FIG. 3, step320). This ends the change detection process.

The video capture interval 228 and delivery delay interval 226 of thepresent invention are suited for use in a calibration capture process232, described for example in commonly owned U.S. patent applicationsSer. No. 12/728,838, and Ser. No. 12/549,011. In such processcalibration test patterns 234, stored in a memory, for example, are usedby the calibration process for projection by projector(s) 102 ontoscreen 104. These projected images are captured by camera 114 and storedin a calibration capture queue 236 for evaluation.

FIG. 4 illustrates a sample timeline calibration capture processhistory. The Timeline vector of FIG. 4 points toward a future time(right) and lists a brief history of past events spanning a time in therecent past (left) up through a current time, Time Now. Events arelisted according to the point in time that they occurred. They includewhen there was a change in the calibration test pattern output 234 ontoa display surface 104 and when video frames were made available bycamera 114. The Available Now frame represents the most recent videoframe available and delivered by the camera in a video capture method.The Unavailable Video block represents data buffers, transferoperations, compression, decompression, or other steps between a TimeNow and when a frame is available to the video capture method. Thesample rate and other characteristics within this interval are unknownbut depicted with a constant frame rate. For example, temporal frameaveraging may occur within this block and the available frame rate tothe video capture method may be lower than the actual device capturerate. For computing delay intervals using a frame counter, it is assumedthat the delivery frame rate from the camera to the video capture methodis constant. The delivery delay interval shown reveals the timenecessary before a change in calibration test pattern will becomeavailable. The video capture interval shown provides a shorter intervalthan the delivery delay interval.

FIG. 5 illustrates an interval detection process history. After severaliterations of dark and light detection patterns, this method hascomputed the shown delivery delay interval and video capture interval.When used by a capture process, display pattern content shall be updatedno faster than the video capture interval 228 to ensure frames containproper capture results. To aid performance, content can be changed anddisplayed at the video capture interval with the initial capturedcontent arriving after a delivery delay interval 226 with followingcontent arriving at each video capture interval thereafter. Forperformance, content displayed should be changed at a rate no slowerthan the delivery delay interval 226. The period from a first contentdisplay to a first associated captured frame at the available now timewill be the delivery delay interval 226 and will mark a content arrivalstart time. The time from a first content display to a second contentdisplay shall be the interval from a content start time to a next usedcaptured frame. This interval shall be no smaller that the video captureinterval 228.

FIG. 6 illustrates a video capture frame change frequency. In FIG. 6, aninterval of time is represented and a top row shows a display signalwith an abrupt change in output pattern from a light value to a darkvalue. Video frames gathered from two different devices capturing thisoutput pattern are presented in a middle and lower row. The middle rowfrom a first device demonstrates quick frame response to the change inpattern with the change fully represented between two successivelycaptured video frames. The device for the lower row, however, requiredfive capture intervals (frame sampling points) before the pattern changewas more closely represented in a captured frame. In this example, thevideo capture interval 228 for the first capture device can be muchshorter than the video capture interval 228 for the second capturedevice. Thus, determining the Delivery Delay Interval 226 and the VideoCapture Interval 228 for a particular image capturing device, e.g.camera, is important to an accurate calibration capture process 232.

Various embodiments can be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations of them.Apparatus can be implemented in a computer program product tangiblyembodied in a machine-readable storage device for execution by aprogrammable processor; and method steps can be performed by aprogrammable processor executing a program of instructions to performfunctions by operating on input data and generating output. Embodimentscan be implemented advantageously in one or more computer programs thatare executable on a programmable system including at least oneprogrammable processor coupled to receive data and instructions from,and to transmit data and instructions to, a data storage system, atleast one input device, and at least one output device. Each computerprogram can be implemented in a high-level procedural or object-orientedprogramming language, or in assembly or machine language if desired; andin any case, the language can be a compiled or interpreted language.Suitable processors include, by way of example, both general and specialpurpose microprocessors. Generally, a processor will receiveinstructions and data from a read-only memory and/or a random accessmemory. Generally, a computer will include one or more mass storagedevices for storing data files; such devices include magnetic disks,such as internal hard disks and removable disks; magneto-optical disks;and optical disks. Storage devices suitable for tangibly embodyingcomputer program instructions and data include all forms of non-volatilememory, including by way of example semiconductor memory devices, suchas EPROM, EEPROM, and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM disks. Any of the foregoing can be supplemented by, orincorporated in, ASICs (application-specific integrated circuits).

While the invention has been described in conjunction with severalspecific embodiments, it is evident to those skilled in the art thatmany further alternatives, modifications and variations will be apparentin light of the foregoing description. Thus, the invention describedherein is intended to embrace all such alternatives, modifications,applications and variations as may fall within the spirit and scope ofthe appended claims.

1. A method for determining a delivery delay interval between projectingan image and delivering the image, comprising: projecting a firstdetection pattern image on a display surface and recording an outputtime, a first video frame time, and a first frame counter value;capturing with a camera a plurality of images including the projectedfirst detection pattern image; comparing each of the plurality ofcaptured images with the first detection pattern image until a match isfound and recording a second video frame time and a second frame countervalue; and determining a delivery delay interval as: the second videoframe time minus the first video frame time, or the second frame countervalue minus the first frame counter value divided by a delivery framerate.
 2. A method as in claim 1 further comprising: identifying a regionof detection within a captured image whose pixels contain a sampleportion of the display surface.
 3. A method as in claim 1, wherein thecomparing comprises: measuring a median among pixel values in the regionof detection within a captured image and maintaining a short movingmedian among successively captured images.
 4. A method as in claim 3further comprising: computing a rate of change between the measuredmedian in successively captured images.
 5. A method as in claim 4further comprising: storing a highest median measured and a lowestmedian measured.
 6. A method as in claim 5 wherein a match is determinedto be found when the rate of change measurement is below a threshold,the median of the currently captured image is nearer to the highest orlowest median measured than to a midpoint between the highest or lowestmedian measured, and the highest and lowest median values are separatedby at least a quarter of a luminance range.
 7. A method as in claim 1further comprising: projecting a second detection pattern that isdifferent than the first detection pattern and determining a deliverydelay interval for the second detection pattern, repeating projectingfirst and second detection patterns and determining a delivery delayinterval for the first and second detection patterns for a plurality ofiterations, and determining a maximum from among the determined deliverydelay intervals, and selecting the maximum as a final delivery delayinterval.
 8. A method as in claim 7 further comprising: after selectingthe final delivery delay interval, projecting a preliminary detectionpattern image on the display surface, waiting the final delivery delayinterval, and then projecting a primary detection pattern image on thedisplay surface and recording an output time, a primary video frametime, and a primary frame counter value; capturing with a camera aplurality of images including the primary detection pattern image;comparing each of the plurality of captured images with the primarydetection pattern image until a match is found and recording a primaryvideo frame time and a primary frame counter value; projecting asecondary detection pattern image that is different from the primarydetection pattern image on the display surface and recording an outputtime, a secondary video frame time, and a secondary frame counter value;capturing with a camera a plurality of images including the secondarydetection pattern image; comparing each of the plurality of capturedimages with a just previously captured image until a change in captureddetection patterns is found and recording a change detection video frametime and a change detection frame counter value at a detection changeoutput time; comparing each of the plurality of captured images with thesecondary detection pattern image until a match is found and recording asecondary video frame time and a secondary frame counter value; anddetermining a change interval as: the secondary video frame time minusthe change detection video frame time or the secondary frame countervalue minus the change detection frame counter value divided by adelivery frame rate.
 9. Apparatus for determining a delivery delayinterval between projecting an image and delivering the image,comprising: a projector that projects an image onto a display surface;the projector projecting a first detection pattern image on the displaysurface and a processor that records an output time, a first video frametime, and a first frame counter value; a camera that captures aplurality of images including the first detection pattern image; and theprocessor further: compares each of the plurality of captured imageswith the first detection pattern image until a match is found andrecords a second video frame time and a second frame counter value; anddetermines a delivery delay interval as: the second video frame timeminus the first video frame time, or the second frame counter valueminus the first frame counter value divided by a delivery frame rate.10. An apparatus as in claim 9 wherein the processor further: identifiesa region of detection within a captured image whose pixels contain asample portion of the display surface.
 11. An apparatus as in claim 9,wherein comparing by the processor comprises: measuring a median amongpixel values in the region of detection within a captured image andmaintaining a short moving median among successively captured images.12. An apparatus as in claim 11 wherein the processor further: computesa rate of change between the measured median in successively capturedimages.
 13. An apparatus as in claim 9 wherein: the projector furtherprojects a second detection pattern that is different than the firstdetection pattern and determines a delivery delay interval for thesecond detection pattern, repeats projecting first and second detectionpatterns and determining a delivery delay interval for the first andsecond detection patterns for a plurality of iterations, and determinesa maximum from among the determined delivery delay intervals, andselects the maximum as a final delivery delay interval.
 14. An apparatusas in claim 13 wherein: after selecting the final delivery delayinterval by the processor, the projector further: projects a preliminarydetection pattern image on the display surface, waits the final deliverydelay interval, and then projects a primary detection pattern image onthe display surface and the processor records an output time, a primaryvideo frame time, and a primary frame counter value; the camera capturesa plurality of images including the primary detection pattern image; theprocessor compares each of the plurality of captured images with theprimary detection pattern image until a match is found and records aprimary video frame time and a primary frame counter value; theprojector projects a secondary detection pattern image that is differentfrom the primary detection pattern image on the display surface and theprocessor records an output time, a secondary video frame time, and asecondary frame counter value; the camera captures a plurality of imagesincluding the secondary detection pattern image; the projector compareseach of the plurality of captured images with a just previously capturedimage until a change in captured detection patterns is found and recordsa change detection video frame time and a change detection frame countervalue at a detection change output time; the projector compares each ofthe plurality of captured images with the secondary detection patternimage until a match is found and records a secondary video frame timeand a secondary frame counter value; and the processor determines achange interval as: the secondary video frame time minus the changedetection video frame time or the secondary frame counter value minusthe change detection frame counter value divided by a delivery framerate.
 15. A tangible computer-readable medium embodying instructionsexecutable by a computer to perform a method of determining a deliverydelay interval between projecting an image and delivering the image, themethod comprising: projecting a first detection pattern image on adisplay surface and recording an output time, a first video frame time,and a first frame counter value; capturing with a camera a plurality ofimages including the projected first detection pattern image; comparingeach of the plurality of captured images with the first detectionpattern image until a match is found and recording a second video frametime and a second frame counter value; and determining a delivery delayinterval as: the second video frame time minus the first video frametime, or the second frame counter value minus the first frame countervalue divided by a delivery frame rate.
 16. A tangible computer-readablemedium as in claim 15, wherein the method further comprises: identifyinga region of detection within a captured image whose pixels contain asample portion of the display surface.
 17. A tangible computer-readablemedium as in claim 15, wherein the method further comprises: measuring amedian among pixel values in the region of detection within a capturedimage and maintaining a short moving median among successively capturedimages.
 18. A tangible computer-readable medium as in claim 17 whereinthe method further comprises: computing a rate of change between themeasured median in successively captured images.
 19. A tangiblecomputer-readable medium as in claim 15 wherein the method furthercomprises: projecting a second detection pattern that is different thanthe first detection pattern and determining a delivery delay intervalfor the second detection pattern, repeating projecting first and seconddetection patterns and determining a delivery delay interval for thefirst and second detection patterns for a plurality of iterations, anddetermining a maximum from among the determined delivery delayintervals, and selecting the maximum as a final delivery delay interval.20. A tangible computer-readable medium as in claim 19 wherein themethod further comprises: after selecting the final delivery delayinterval, projecting a preliminary detection pattern image on thedisplay surface, waiting the final delivery delay interval, and thenprojecting a primary detection pattern image on the display surface andrecording an output time, a primary video frame time, and a primaryframe counter value; capturing with a camera a plurality of imagesincluding the primary detection pattern image; comparing each of theplurality of captured images with the primary detection pattern imageuntil a match is found and recording a primary video frame time and aprimary frame counter value; projecting a secondary detection patternimage that is different from the primary detection pattern image on thedisplay surface and recording an output time, a secondary video frametime, and a secondary frame counter value; capturing with a camera aplurality of images including the secondary detection pattern image;comparing each of the plurality of captured images with a justpreviously captured image until a change in captured detection patternsis found and recording a change detection video frame time and a changedetection frame counter value at a detection change output time;comparing each of the plurality of captured images with the secondarydetection pattern image until a match is found and recording a secondaryvideo frame time and a secondary frame counter value; and determining achange interval as: the secondary video frame time minus the changedetection video frame time or the secondary frame counter value minusthe change detection frame counter value divided by a delivery framerate.